Skip to content

Require an AI cross-review before merge - #229

Open
lwwmanning wants to merge 16 commits into
mainfrom
wm/vortex-cross-ai-review-f2ab62
Open

lwwmanning wants to merge 16 commits into
mainfrom
wm/vortex-cross-ai-review-f2ab62

Conversation

@lwwmanning

Copy link
Copy Markdown
Contributor

What changes

This adds a merge gate for AI review to both sites' pull requests, in a new workflow, ai-review.yml. The other checks stay as they are.

Before, Codex reviewed some pull requests on its own, and Claude reviewed only when someone tagged @claude. Neither review blocked a merge. After, the gate sets the commit status AI cross-review on the head commit. Every pull request needs a clean Codex review of that commit. If the footer or the author names Codex, the pull request also needs a clean Claude review, which the workflow runs.

A P0 or P1 finding fails the gate, so auto-fix can act on it. When Codex has not reviewed the head commit, the gate posts @codex review. Renovate and Dependabot pull requests pass without review.

After merge, the main ruleset must require AI cross-review before the gate blocks anything.

How it was checked

bun test scripts/ai-review-gate.test.ts, the typecheck and bun run check:ci pass. Dry runs against #216, #220, #225, #206 and #227 gave the expected status for each.

  • Every commit is signed off (git commit -s), for the DCO check.
  • bun run check:ci passes.
  • Each new image or other asset has an annotation in REUSE.toml.

🤖 Generated with Claude Code

Every pull request needs a Codex review of its head commit. A pull request
whose footer or author names Codex also needs a Claude review. The gate sets
the commit status "AI cross-review" for the main ruleset to require.

Signed-off-by: Will Manning <will@willmanning.io>
@vercel

vercel Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
vortex.dev Ready Ready Preview Nov 21, 2026 3:37am UTC

Request Review

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

Comment thread .github/workflows/ai-review.yml Fixed
Comment thread .github/workflows/ai-review.yml Fixed
…view-f2ab62

Signed-off-by: Will Manning <will@willmanning.io>

# Conflicts:
#	CLAUDE.md
@lwwmanning
lwwmanning marked this pull request as ready for review November 21, 2026 01:51
CodeQL flagged the explicit checkout of the head commit as an untrusted
checkout, because the workflow also runs on issue_comment.

Signed-off-by: Will Manning <will@willmanning.io>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: 🔄 Running

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-11-21T03:37:54.231921Z f0e8231 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

CodeQL flags any job that runs the pull request's code in a workflow that
also runs on issue_comment. ai-review.yml now runs on pull_request only,
and ai-review-recheck.yml reruns main's copy of the gate on comments and
reviews. The gate's token also gets pull-requests: write, which a comment
on a pull request needs.

Signed-off-by: Will Manning <will@willmanning.io>
@github-actions

Copy link
Copy Markdown

@codex review

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

🔦 Lighthouse audit

URL Perf A11y Best practices SEO
/ 🟢 100 🟢 100 🟢 100 🟢 100
/bench 🟢 100 🟢 100 🟢 100 🟢 100
/bench/chart/ct.eyJrIjoiQ29tcHJlc3Npb25UaW1lIiwiZGF0YXNldCI6ImFyYWRlIiwiZGF0YXNldF92YXJpYW50IjpudWxsfQ 🟢 100 🟢 100 🟢 100 🟢 100
/blog 🟢 100 🟢 100 🟢 100 🟢 100
/blog/btrblocks-compressor 🟢 99 🟢 100 🟢 100 🟢 100

Run 44862108226 · 2026-11-21 03:39 UTC

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e51dd80774

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ai-review.yml Resolved
Comment thread scripts/ai-review-gate.ts Resolved
Comment thread scripts/ai-review-gate.ts Resolved
Comment thread .github/workflows/ai-review.yml Resolved
Comment thread .github/workflows/ai-review.yml Resolved
Comment thread scripts/ai-review-gate.ts Resolved
Comment thread .github/workflows/ai-review.yml Resolved
Codex's review found that the gate ran in the pull request's context: forks
and Dependabot PRs got a read-only token, a PR could change the script that
judged it, and fork PRs could never get a Claude review. The workflow now
runs on pull_request_target, comments and reviews, and every job checks out
main. Claude reads the PR through the API.

The gate also asks Codex again after 20 minutes without a start (three times
at most), retries a Claude review that ended without a verdict, and ignores
dismissed Codex reviews. ci.yml runs the gate's tests.

Signed-off-by: Will Manning <will@willmanning.io>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 25fbcceb11

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/ai-review-gate.ts Resolved
Comment thread scripts/ai-review-gate.ts Resolved
Comment thread scripts/ai-review-gate.ts Resolved
The gate sets itself to pending before it reads the reviews, so a failed
run cannot leave an old success in place. It fails when another open PR
has the same head commit, because a commit status counts for every PR on
that commit. A submitted Codex review of the head now counts even while
the summary comment still says the review is running.

Signed-off-by: Will Manning <will@willmanning.io>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 20e6373921

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/ai-review-gate.ts Resolved
Comment thread scripts/ai-review-gate.ts Resolved
The Claude status name now holds the PR number, so a review of one PR does
not count for another PR on the same commit. The gate counts Claude's
blocking findings from its unresolved P0 and P1 threads, as it does for
Codex, so resolving a thread and commenting clears it without a new push.

Signed-off-by: Will Manning <will@willmanning.io>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c5093392b2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/ai-review-gate.ts Resolved
Comment thread scripts/ai-review-gate.ts Resolved
Comment thread scripts/ai-review-gate.ts Resolved
Comment thread scripts/ai-review-gate.ts Resolved
A PR that moves to another base branch keeps its head commit, but its diff
can change, so the gate posts a marker and fails until a new commit
arrives. Only P0 and P1 threads count as evidence for Claude's counted
findings. The gate reads every page of the commit's statuses, and counts
only marker comments from github-actions[bot].

Signed-off-by: Will Manning <will@willmanning.io>
…view-f2ab62

Signed-off-by: Will Manning <will@willmanning.io>

# Conflicts:
#	CLAUDE.md

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3516cffac1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/ai-review-gate.ts Resolved
Comment thread .github/workflows/ai-review.yml Resolved
Comment thread scripts/ai-review-gate.ts Resolved
Comment thread .github/workflows/ai-review.yml Resolved
Comment thread .github/workflows/ai-review.yml Resolved
Drop the pull_request_review trigger, which runs the PR's copy of the
workflow. Claude reviews only PRs by authors with write access; an
outside PR that Codex helped write needs a maintainer's approval of the
head commit. Record a base change as a commit status before the notice
comment. Mark a Claude review as started only after setup, and retry a
review that stays pending for 45 minutes. Check at verdict time that
each blocker Claude counts has an open thread.

Signed-off-by: Will Manning <will@willmanning.io>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d46d93d67d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ai-review.yml Resolved
Comment thread scripts/ai-review-gate.ts Resolved
Comment thread .github/workflows/ai-review.yml Resolved
author_association calls a read-only organization member a MEMBER, so
the gate now asks the permission API whether the author, or an approver
of the head commit, can push. The Claude action can check out the PR's
branch, so the job checks out main again before it records the verdict,
and Claude reads CLAUDE.md from main. The gate's rerun after a review
is now a job in the gate's concurrency group.

Signed-off-by: Will Manning <will@willmanning.io>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 576ce296d9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/ai-review-gate.ts Resolved
Comment thread .github/workflows/ai-review.yml Resolved
Comment thread scripts/ai-review-gate.ts Resolved
Comment thread scripts/ai-review-gate.ts Resolved
…odex's finding flag

The first gate run on a head commit records the base branch as a status,
and later runs compare against it, so a cancelled waiting run cannot lose
a base change. An approval counts only while it is the reviewer's latest
approval or request for changes. When Codex reports findings on the head
commit but no Codex thread on that commit shows one, the gate fails.

Signed-off-by: Will Manning <will@willmanning.io>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 92517a3682

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ai-review.yml Resolved
A commit status belongs to the commit, so another PR with the same head
would inherit it. The gate now runs on closed and sets its status back
to pending unless the PR merged.

Signed-off-by: Will Manning <will@willmanning.io>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d313911250

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ai-review.yml Resolved
Comment thread scripts/ai-review-gate.ts
Comment on lines +388 to +391
const shown = input.threads.some(
(t) => isCodexThread(t) && t.commitSha === input.headSha
);
if (reported && !shown) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Detect partially missing Codex finding threads

When Codex reports multiple findings but only one inline thread is published, shown becomes true even if another missing finding is P0/P1. The fresh partial-publication case is not covered by the prior no-thread fix: the visible thread can be resolved or P2/P3, after which the gate publishes success while the missing blocker has no resolvable thread. Preserve enough structured finding information to detect incomplete publication rather than accepting any single thread as evidence for all findings.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed. Codex's summary gives only booleans (has_any_findings, has_unresolved_findings), not a count, so the gate cannot tell how many threads to expect. Codex posts its inline comments as part of one review: on this PR, each Codex review holds all of its comments (from 1 to 7). That suggests a review is published whole or not at all, which the no-thread check covers. I'm leaving this thread open for a maintainer to decide.

🤖 Addressed by Claude Code

Comment thread scripts/ai-review-gate.ts Resolved
Comment thread .github/workflows/ai-review.yml Resolved
Comment on lines +56 to +60
permissions:
contents: read
# Set the `AI cross-review` status.
statuses: write
# Post `@codex review` when Codex has not reviewed the head commit. A

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Prevent pull-request workflows from forging the gate status

The required result is a generic commit status written by the GitHub Actions app, so a same-repository contributor can modify an existing pull_request workflow such as ci.yml or lighthouse.yml, request statuses: write, and POST AI cross-review: success for their own head SHA without satisfying this gate. GitHub documents that pull_request workflows run against the PR merge branch and that workflow authors can change GITHUB_TOKEN permissions, including statuses: write; pinning the required status to GitHub Actions therefore does not distinguish the trusted base workflow from PR-controlled workflows. Publish the result through a dedicated app/source that PR workflows cannot impersonate, or enforce this as a required workflow instead. GitHub pull-request event semantics GitHub token permissions

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed. This is a real limit of a commit status: in this repository, any pull_request workflow that a writer edits could post one. Fork PRs get a read-only token, so they cannot. The fix is to publish the result from a dedicated GitHub App, and pin the ruleset's required check to that app. Creating an app is a setup step for a maintainer, so I'm leaving this thread open for a maintainer to decide.

🤖 Addressed by Claude Code

Comment thread scripts/ai-review-gate.ts Resolved
The gate job sets its status to pending before checkout and setup, so a
failed setup cannot leave an earlier pass. A closed PR, merged or not,
withdraws its status. The Claude job records a verdict only when main
was checked out again. The base record holds the branch name's SHA-256,
so a long name cannot be cut to match another.

Signed-off-by: Will Manning <will@willmanning.io>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 37b41373b0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ai-review.yml Resolved
Comment thread scripts/ai-review-gate.ts Resolved
Comment thread scripts/ai-review-gate.ts Resolved
A review, or a PR closing beside a sibling with the same head, starts
no gate run. Every 15 minutes a scheduled run from main evaluates each
open, ready PR again under the gate's lock, and writes the status only
when the result changed. A Codex review still in progress after an hour
counts as stopped, so the gate may ask for it again.

Signed-off-by: Will Manning <will@willmanning.io>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cc31d561a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ai-review.yml Resolved
Comment thread scripts/ai-review-gate.ts Resolved
…reviews

The refresh starts the workflow by workflow_dispatch for a PR whose
Claude review is due, because only a gate run can start the Claude job.
The gate starts at most three Claude reviews per commit. A dispatched
run has no PR context for the inline comment tool, so Claude can post
findings through the review comments API. A dismissed Codex review of
the head commit no longer counts through Codex's summary comment.

Signed-off-by: Will Manning <will@willmanning.io>

This branch was successfully deployed

1 active deployment
Preview — f0e8231f Deployed Nov 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants